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(57) Abstract 



A method for generating digital watermarks and for exchang- 
ing data containing such watermarks is described. It is based on a 
watermarking technique which is robust against image transformation 
techniques such as compression, rotation, translation, scaling and/or 
change of proportion. It uses modulation of the magnitude compo- 
nents in Fourier space and adds/reads a template. in the log-polar 
or log-log transform of the magnitude components. The template is 
used for analyzing scaling and rotation or change of proportion. In 
addition, the system applies cryptographic protocols and public key 
techniques for both, encoding the watermark and transferring wa- 
termarked data. Preferably, an author (CH) encodes the watermark 
using an asymmetric cryptographic key pair provided by a public key 
infrastructure (PKI) and registers the watermarked data at a trusted 
registration party (CCC) before transmitting the data to a receiving 
party (B). The latter can use the public key infrastructure (I) for ver- 
ifying autorship. Data exchanged by the parties are encrypted using 
the cryptographic keys. In addition, image (video) originality verifi- 
cation is supported by the same asymmetric key pair as for content 
protection and for copyright protection. 
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Method for generating and verifying digital watermarks 
and for exchanging data containing digital vratermarks 

5 Cross References to Related Applications 

This application claims the priority of Euro- 
pean patent application 97810708.4, filed Sept. 26, 1997, 
the disclosure of which is incorporated herein by refer- 
10 ence in its entirety. 

Technical Field 

The present invention relates to methods for 
15 generating and verifying digital watermarks and for 

transmitting data containing digital watermarks according 
to the preamble of the independent claims. 

Background Art 

20 

Digital watermarking is a method for marking 
data sets, such as images, sound or video. A digital wa- 
termark consists of a slight modification of the data set 
that does not affect the data set's usability but that 
25 can be detected using dedicated analysis software or ap- 
paratus. Watermarking can e.g. be used for marking 
authorship or ownership of a data set. It can also be ap- 
plied for verifying the originality of the multimedia 
data content, where the loss of originality refers to the 
30 degree of contents modification suffered by the image. 

Digital watermarking can be seen as a funda- 
h mental-problem in digital communications (see e.g. I. 

Cox, J. Killian, T. Leighton, and T. Shamoon, "Secure 
1 spread spectrum communication for multimedia" , Proceedings 

35 of the IEEE International Conference on Image Processing, 
Lausanne, Switzerland, September 1996) . Early methods of 
encoding watermarks consisted of no more than increment- 
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ing an image component to encode a binary '1' and decre- 
menting to encode a *0' (G. Caronni "Assuring Ownership 
Rights for Digital Images" in H . H. Brueggemann and W. 
Gerhardt-Haeckl, editors, Reliable IT Systems VIS ^95, 
5 vieweg Publishing Company, Germany, 1995). Tirkel et al . 
(A. Z. Tirkel, G- A. Rankin, R. G. van Schyndel, W. J. 
Ho, N. R . A. Mee, and C. F. Osborne, "Electronic water- 
mark", in Dicta-93, pages 666-672, Macquarie University, 
Sydney, December 1993) and van Schyndel et al . (A. Z. 

10 Tirkel, R. G. van Schyndel, and C. F . Osborne, tt a two- 
dimensional digital watermark", in ACCV'95, pages 378- 
383, University of Queensland, Brisbane, December 6-8 
1995) have applied the properties of m-sequences to pro- 
duce oblivious watermarks resistant to filtering, crop- 

15 ping and reasonably robust to cryptographic attack. Ma- 
tsui and Tanaka (K. Matsui and K. Tanaka, "Video- 
Steganography : How to secretly embed a signature in a 
picture", in IMA Intellectual Property Project Proceed- 
ings, pages 187-2 06, January 1994) have applied linear 

20 predictive coding for watermarking . Their approach to 
hiding a watermark is to make the watermark resemble 
quantization noise. Tirkel and Osborne (see above) were 
the first to note the applicability of spread spectrum 
techniques to digital image watermarking. Since then 

25 there has been an increasing use of spread spectrum in 
digital watermarking. It has several advantageous fea- 
tures, such as cryptographic security (see Tirkel and Os- 
borne, above), and is capable of achieving error free 
transmission of the watermark near or at the limits given 

30 by the maximum channel capacity (J. Smith and B. Co- 

miskey, "Modulation and information hiding in images", in 
Ross Anderson, editor, Proceedings of the First Interna- 
tional Workshop in Information Hiding, Lecture Notes in 
Computer Science, pages 2 07-22 6, Cambridge, UK, May /June 

35 1996. Springer) . Fundamental information theoretic limits 
to reliable communication have been discussed by some 
authors (see Smith and Comiskey, above) . The shorter the 
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payload of a watermark, the better are the chances of it 
being communicated reliably. Spread spectrum is an exam- 
ple of a symmetric key cryptosystem (B. Schneier, "Ap- 
plied Cryptography" , Wiley, 2nd edition, 1995). System 
5 security is based on proprietary knowledge of the keys 

(or pseudo random seeds) which are required to embed, ex- 
tract or remove an image watermark. One provision in the 
use of a spread spectrum system is that it is important 
that the watermarking be non-invertible because only in 

10 this way can true ownership of the copyright material be 
resolved (S. Craver, N. Memon, B. Yeo, and M. Yeung, Tan 
invisible marks resolve rightful ownership's ?", 
IS&T/SPIE Electronic Imaging' "91 : "Storage and Retrieval 
of Image and Video Databases", 1997). 6 Ruanaidh et al . 

15 (J. K. 6 Ruanaidh, W. J. Dowling, and F. M. Boland, 

"Phase watermarking of images", IEEE International Con- 
ference on Image Processing, Lausanne, Switzerland, Sep- 
tember 1996) and Cox et al . (see above) have developed 
perceptually adaptive transform domain methods for water- 

20 marking. In contrast to previous approaches listed above 
the emphasis was on embedding the watermark in the most 
significant components of an image or a video frame. The 
general approach used in these papers is to divide the 
image into blocks. Each block is mapped into the trans - 

25 form domain using either the Discrete Cosine Transform 
(W. B. Pennebaker and J. L. Mitchell, "JPEG Still Image 
Compression Standard", Van Nostrand Reinhold, New York, 
1993), the Hadamard Transform (W. G . Chambers, "Basics of 
Communications and Coding", Oxford Science Publications. 

30 Clarendon Press Oxford, 1985) or the Daubechies Wavelet 
Transform (W.H. Press, S.A. Teukolsky, W.T. Vetterling, 
and B.P. Flannery, "Numerical Recipes in C", Cambridge 
University Press, second edition, 1992) . The phase compo- 
nent of the image or video frame is then modified accord- 

3 5 ing to the pseudo-random sequence containing the water- 
marking information". 
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Information can be embedded using the DCT (J. 
j. K. 6 Ruanaidh, w. J * Dowling, and F. M. Boland, "Wa- 
termarking digital images for copyright protection", IEEE 
Proceedings on Vision, Image and Signal Processing, 
5 143(4) : 250-256, August 1996, based on the paper of the 
same title at the IEEE Conference on Image Processing and 
Its Applications, Edinburgh, July 1995) FFT magnitude, 
and phase, Wavelets (see refs. of Ruanaidh, Dowling and 
Boland, above) , Linear Predictive Coding (see Matsui et 

io al., above) and fractals (P. Davern and M. Scott, "Frac- 
tal based image s teganography " , in Ross Anderson, ed., 
Proceedings of the First International Workshop in Infor- 
mation Hiding, Lecture Notes in Computer Science, pp. 
279-294, Cambridge, UK, May/June 1996. Springer Verlag) . 

15 The key to making watermarks robust has been 

the recognition that in order for a watermark to be ro- 
bust it must be embedded in the perceptually significant 
components of the image (see ref . of Ruanaidh, Dowling 
and Boland, and ref. of I. Cox, J. Killian, T. Leighton, 

20 and T. Shamoon above) . Objective criteria for measuring 
the degree to which an image component is significant in 
watermarking have gradually evolved from being based 
purely on energy content (see refs. of Ruanaidh et al . , 
Cox et al. above), to statistical (see I. Pitas, "A 

25 method for signature casting on digital images", Proceed- 
ings of the IEEE International. Conference on Image Proc- 
essing, Lausanne, Switzerland, September 1996) and psy- 
chovisual (see J.F. Delaigle, C. De Vleeschouwer , B. 
Macq, "Digital watermarking", Proceedings of the SPIE 

30 Electronic Imaging: Science and Technology, vol. 2659: 
Optical Security and counterfeit Deterrence Techniques, 
San Jose, February 1996 and M.D. Swanson, B. Zhu and A. 
Tewfik, "Transparent robust image watermarking", Proceed- 
ings of the IEEE International Conference on Image Proc- 

35 essing, Lausanne, Switzerland, September 1996). 

The industrial importance of digital water- 
marking has resulted in a number of products on the mar- 
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ket, either based on spread spectrum techniques or addi- 
tional registration services . They include the Picture- 
marc system by Digimarc (RHOADS, B. Geoffrey, Digimarc 
Corp (US) , "Steganography Systems, WO 96/36163 A, Sure- 
5 Sign (former FBI's Fingerprint) by HighWater Signum (WO 
96/27259), IP2 system by Intellectual Protocols, the Ar- 
gent system by Digital Information Commodities Exchange, 
the PixelTag system by the MIT Media Lab, the SysCop sys- 
tem from Zhao and Koch by the Frauenhof er-Ins titut fur 

10 Graphische Datenverarbei tung (J. Zhao and E. Koch, "Em- 
bedding robust labels into images for copyright protec- 
tion" , Proceedings of the International Congress on In- 
tellectual Property Rights For Specialized Information, 
Knowledge and New Technology, August 1995 J. Zhao, "A WWW 

15 Service To Embed And Prove Digital Copyright Watermarks", 
Proc . Of the European Conference on Multimedia Applica- 
tion, Services and Techniques, vol. 2, Louvain-La-Neuve , 
Belgium, May 19 96) , and the Tigermark system from NEC 
(European patent Application EP 766468A, Nippon Electric 

20 Corporation (NEC), April 1997) 

The approach proposed by Digimarc (see WO 
96/36163) adds or subtracts small random quantities from 
each pixel according to the least significant bit of each 
pixel compared with the binary mask. The originality of 

25 their approach consists in the use of "subliminal digital 
graticules" that will help in recovering a rotation R and 
a scaling S performed on the marked image. They use an 
exhaustive search strategy based on these graticules to 
recover R and S. This stands in contrast to the template 

30 embodiment described here, where the use of log-polar or 
log-log mapping of the Fourier transform of the image 
combined with cross-correlation in the log-polar or log- 
log plane avoid such a search. 

The Highwatez? approach (WO 96/2725?) describe 

35 a permutation technique to modify the values of the data 
elements according to certain rules which depend on the 
message . 
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6 

The approach of Zhao and Koch, based on the 
JPEG image compression algorithm, . proceeds by segmenting 
the image into individual 8x8 blocks. Only eight coef- 
ficients occupying particular positions in the 8x8 
5 block of DCT coefficients can be marked. These comprise 
the low frequency components of the image block but ex- 
clude the mean value coefficient as well as the low fre- 
quencies. Three of the remaining DCT coefficients are se- 
lected using a pseudo random number generator to convey 

10 information. The resemblance of this technique to fre- 
quency hop spread spectrum communications is also men- 
tioned and the blocks are placed at random positions in 
the image. A WWW registration service has been proposed 
for a local registration and a local watermarking, for a 

15 server registration and a server watermarking, and for a 
local watermarking and a server registration. The ap- 
proach is based on a trusted third party model (WWW 
server and Watermark Embedding Gateway) . This model re- 
quires from the Copyright Holder the transfer of relevant 

20 confidential information applied for the watermarking 

process. It is, therefore, possible that the owner of the 
trusted third party system may impersonate the Copyright 
Holder and infringe his copyright. Since the applied key 
for the embedding is not a cryptographic key, copyright 

25 protection and communication security are addressed by 

two different technical solutions, namely the SysCop sys- 
tem and the s-http protocol. These two technical solu- 
tions are applied independently. There is no third party 
ver ification procedure supported which allows the verifi- 

3 0 cation of the seed, applied for the embedding of the wa- 
termark, by independent parties, such as a court of law. 
The s-http protocol (SSL security protocol) differs from 
the protocol presented below in many aspects (for exam- 
ple, the non-repudiation, security service is not sup- 

35 ported by the s-http protocol) . The keys applied for the 
embedding of the mark are furthermore not registered in 
the SysCop system. For copyright verification, the Copy- 
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right Holder has to disclose his key. The information 
generated by the trusted third party is based on the 
cover data, but not on the stego data. 

I. Cox et al from NEC (see EP 7 66 468, above) 
5 propose to insert watermark into the perceptually sig- 
nificant components of a decomposition of the data in a 
manner so as to be visually imperceptible. In contrast to 
the method described here, they need the original data 
which is compared to the watermarked data to obtain an 

10 extracted watermark. 

J.-F. Delaigle at al . (J.-F. Delaigle, J.-M. 
Boucqueau, J. -J. Quisquater and B. Macq, "Digital Images 
protection techniques in a broadcast framework: An over- 
view", Proceedings of the European Conference on Multime- 

15 dia applications, Services and Techniques, vol. 2, Lou- 
vain-La-Neuve, Belgium, May 1996, J.-F. Delaigle, C. De 
Vleeschouwer & B. Macq, "Digital Watermarking", Proceed- 
ings of the SPIE, vol. 2 659, 1 February 1996) have ap- 
plied signature labeling techniques for the copyright 

20 protection of digital images. The approach presented is 
very similar a EDI security standards. The labeling does 
not influence the multimedia data. Their approach is 
based on an enhanced image format and generates a digital 
signature label in front of the image. This signature la- 

25 bel can be easily overwritten or destroyed. The registra- 
tion entity supports no secure on-line communication pro- 
tocol and is constrained by a legal trusted third party. 
In addition, no means are provided to resolve a conflict 
if multiple watermarks have been embedded in the same im- 

30 age. In an enhanced architecture they propose a general 
watermarking function which uses the output of a hash 
function as the payload of the watermark. This watermark 
function does not support third party verification and is 
not based on a spread spectrum technique. In addition, 

3 5 different types of watermarks are not supported. The 

masking scheme presented depend on a ciphering function 
for the inscription. In contrast to the approach pre- 
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sented in this disclosure, the secret key has to be re- 
vealed for copyright verification and no coding/decoding 
along with cryptographic digital signatures are applied, 
in addition, the cryptographic key applied is only used 
5 for ciphering and not for other functional purposes rele- 
vant for copyright protection as defined in this disclo- 
sure . 

S. Matyas at all (Stephen M. Matyas, Donald 
B. Johnson, An V. Lee, Rostislaw Prymak, William C. Mar- 
io tin, William S. Rohland, and John D. Wilkins, "EP 0 534 
419 A", Stephen M. Matyas, Donald B. Johnson, An V. Lee, 
Rostislaw Prymak, William C. Martin, William S. Rohland, 
and John D. Wilkins, "EP 0 539 726 A") have specified a 
system which is based on an architecture with two differ- 
15 ent entities, namely the data processor with a crypto- 
graphic system and the network certification center. The 
overall system security depends on a hierarchical crypto- 
graphic key scheme and digital certificates are only gen- 
erated for a specific data set, called control vectors. 
These control vectors set up the basis to identify the 
access rights of users and associated processes they have 
initiated. The main focus of the specified system is the 
enforcement of a dedicated security policy which is based 
on a hierarchical role model. The system is based on a 
25 hardware based security processors and applies symmetric 
and asymmetric cryptographic keys . The cryptographic pro- 
tocols applied are different to the protocols presented 
in this disclosure. The emphasis is to provide a method 
for controlling the use of private and public keys which 
30 is not the purpose of our system. In addition, one entity 
needs several different types of keys (symmetric and 
asymmetric) in contrast to our approach which uses for 
one entity one asymmetric key pair only. 

Tanaka et al. (K. Tanaka and K. Matsui, "A 
35 Digital Signature scheme on a Document for MH Facsimile 
Transmission", Electronics & Communications in Japan, 
Part I - Communications, Vol. 74, No. 8, August 1991) 



20 
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propose a digital signature scheme for watermarking fac- 
simile documents (binary images) . This scheme modify the 
length of certain runs of data with a single bit of the 
signature data. 

5 

Disclosure of the Invention 

It is an object of the present invention to 

10 provide a system of the type mentioned above that pro- 
vides a simple and secure way of generating and transmit- 
ting watermarked data. This object is achieved by the 
methods described in the claims. 

In one aspect of the invention, this object 

15 is achieved by an integrated solution method for generat- 
ing and transmitting a data set between two parties H and 
B comprising the steps of a) providing a cover data set 
corresponding to the data set to be transmitted, b) gen- 
erating a stego data set of said cover data set by 

20 embedding at least one digital watermark in said cover 
data set, wherein said watermark is encoded using at 
least one key of an asymmetric cryptographic key pair of 
H, said key pair comprising a secret private key and a 
known public key derived therefrom, and c) encrypting 

25 said stego data set using said key pair of H, . d) trans- 
mitting said encrypted stego data set from said party H 
to said party B. 

The party creating the watermark can embed a 
detection, a private and a public watermark in the data 

30 set, wherein the detection or the private watermark is 
derived from the private key, the public watermark from 
the public key. The public watermark can be detected by 
third parties while the private watermark can only be de- 
tected using private information. Preferably, the detec- 

35 tion or private watermark is not derived from the private 
key directly but from a hash value of the same and/or 
from a signature generated with the same, such that the 
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10 

author of the watermark does not have to reveal his pri- 
vate key if the private watermark is to be verified. 

In another aspect of the invention, the cover 
data set is provided with a digital watermark and derived 
5 stego data then securely transmitted to a registration 
party that permanently stores at least time information, 
origin of the stego data set, and a digital copyright 
certificate . 

In another aspect of the invention, a tem- 

10 plate modulation pattern is added to the Fourier trans- 
form of an image that is to be provided with a watermark. 
For checking the watermark, the Fourier transform of the 
stego-image is calculated. From this Fourier transform, 
the log-polar mapping transform is generated, which is 

15 then searched for the modulation pattern. Using the log- 
polar transform of the Fourier transform has the advan- 
tage that scaling and rotation of the stego-image are ex- 
pressed in translations. This allows an easy search for 
rotation and scaling using cross-correlation techniques. 

20 However, especially for video data, a change 

of proportion (different horizontal and vertical scaling) 
is more probable than a rotation. In such cases, the tem- 
plate modulation pattern is rather searched in the log- 
log transform of the Fourier transform. Similarly to the 

25 log-polar map, the log-log map allows to express the 

horizontal scaling and vertical scaling in translations 
and cross-correlation techniques can be applied to search 

the template. 

In still another aspect of the invention, the 

3 0 image to be watermarked is divided into blocks and the 
magnitude components of the Fourier transform of each 
block is modulated using the same pattern. This method 
provides robustness against cropping of the stego-image 
because a cropping leads, to a circular translation in 

35 each block. Preferably, the magnitude components of the 
Fourier transform are modulated, wherein the sign of the 
modulation should be derived from the phase components, 
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thereby reducing interference between the image data and 
the watermark as explained in the following disclosure. 

In a further aspect, the invention consists 
of a method for generating and transmitting a data set 
5 between two parties H and B comprising the steps of pro- 
viding a cover data set corresponding to the data set to 
be transmitted, generating a stego data set of said cover 
data set at a party H by generating at least one digital 
watermark in said cover data set, transmitting a has 

10 value of said stego data set to a registration party, and 
permanently storing certification data at said registra- 
tion party, said certification data comprising said hash 
value of said stego data set, a digital time stamp and 
information designating said party H. 

15 In a further aspect, the invention relates to 

a method for generating a stego data set from a cover 
data set by adding a watermark to said cover data set 
comprising the steps of dividing said stego data sets 
into blocks, calculating a lapped orthogonal transform of 

20 each of said blocks, and applying said watermark to said 
lapped orthogonal transforms. 

In another aspect, the invention relates to a 
method for generating a watermark in a cover data set 
(CD) representing a two or three dimensional data set, 

25 especially for step b) of one of the preceding claims, 

comprising the following steps: A) generating a template 
modulation pattern (T') using a random number generator 
seeded by a key (K) , B) calculating the Fourier transform 
of at least part of said cover data set (CD) for generat- 

30 ing Fourier components, of said cover data set, C) modu- 
lating at least part of said Fourier components using 
said template modulation pattern (T' ) , D) using the in- 
verse Fourier transform for generating a stego-image 

The invention further relates to a method for 

35 verifying a watermark in a possibly rotated and/or scaled 
version of a two or three dimensional stego data set, 
comprising the steps of: A) calculating a Fourier trans- 
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12 

form of said stego data set (SD) , B) calculating a log- 
polar or a log-log transform of said Fourier transform of 
said stego data set, and C) calculating the correlation 
between said log-polar or log-log transform and a tem- 
5 plate (T) , which template is the log-polar or log-log 
transformation of said watermark. 



Brief Description of the Drawings 



10 



The invention will be better understood and 
objects other than those set forth above will become ap- 
parent when consideration is given to the following de- 
tailed description thereof. Such description makes refer- 
15 ence to the annexed drawings, wherein: 

Fig. 1 the parties involved in individual wa- 
termark protection, 

Fig. 2 the parties involved in watermark pro- 
tection using registered cryptographic keys, 
20 Fig / 3 the parties involved in watermark pro- 

tection using registered cryptographic keys and a regis- 
tration party, 

Fig. 4 the steps taken for embedding a water- 
mark, 

25 Fig. 5 the steps for generating the template, 

Fig. 6 the steps for reading a watermark, 
Fig. 7 the steps for reading the template, 
Fig. 8 the steps for embedding watermark in a 
rotation, scale and translation invariant domain, 
30 Fig. 9. the steps for embedding the watermark 

in an image avoiding to map the original image into the 
rotation, scale and translation invariant domain, 

Fig. 10. the steps to extract the watermark 
from the image, 
35 Fig. 11 the tiling of the watermark in a 

stego-image or stego video frame , and 
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Fig. 12 the tiling of the watermark in a 
cropped stego-image or cropped stego video frame. 

Modes for Carrying Out the Invention 

I. Terms and Symbols: 

Before describing a preferred method and ap- 
paratus according to the invention, some key terms and 
symbols used in its description are explained in the fol- 
lowing : 

"Image": An image in either digital or physi- 
cal form which may constitute a still image or a video 
frame. It can also refer other types of data, such as 
video and sound, in particular when being used within the 
context of the protection and owner authentication meth- 
ods of section II of the disclosure. 

" Signal " : A signal in either digital or 
physical form. It may refer to one dimensional or multi- 
dimensional signals such as image and video. 

" Copyright Holder (CH) " : A party (or a proc- 
ess acting on behalf of it) "owning" a digital image or 
video. This is the party that generates the watermarks. 

" Buyer (B) " : A party (or a process acting on 
behalf it) which obtains (e.g. by purchase) via elec- 
tronic means a specific' image from the CH . 

" Stego " : Implies that an image or video data 
is marked. The stego image is also referred to as the 
stego data set (e.g. stego image or video frame). 

" Cover " : Implies that an image or data is un- 
marked. The cover image is also referred to as the cover 
data set (e.g. cover image or video frame). 

" Watermark " : The form the IAD takes- when it 
is in a form suitable for embedding in a signal. 

" Copyright Certificate Center (CCC) " : An or- 
ganization (or a process which acts on behalf of it) 
which registers copyright ownership for a specific image 
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or video.. Successful registration is only based on a 
sender verification procedure. After successful registra- 
tion a digital copyright certificate can be generated. 
The CCC does not act as trusted third party in our sys- 
tem. 

" Digital copyright certif icate " : Digital 
copyright data which comprise the copyright certificate 
data and a digital signature. 

" Copyright Request Data (CRD) ": Copyright 
data which contains the stego-image, the image ID of the 
cover-image, a Universal Copyright Convention Notice, a 
Copyright Symbol, the term ' 'Copyright' ' , the year of the 
copyright, the name of the copyright holder, and the 
phrase ' 'All Rights Reserved' 1 . 

" Copyright Certificate Data, (CCD) " : Copyright 
data which contains relevant copyright information. 

" Digital signature " : A data string which has 
been generated by a cryptographic digital signature gen- 
eration transformation . 

" Digital signature generation transforma- 
tion " : A method for producing a digital signature. 

" Digital signature verification transforma- 
tion " : A method for verifying whether a digital signature 
is authentic or not. 

" Digital signature scheme " : A scheme based 
on asymmetric cryptographic techniques whose private 
transformation is used for the digital signature genera- 
tion and whose public transformation is used for the 
digital signature verification. 

" Digital signature scheme with message recov- 
ery " : A digital signature scheme for which a priori 
knowledge of the input data is not required for the sig- 
nature verification transformation. 

" Digital signature scheme with appendix " : A 
digital signature scheme for which the input data is re- 
quired as input to the digital signature verification 
transformation . 
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" Asymmetric key pair ": A pair of related 
cryptographic keys where the private key defines the pri- 
vate transformation and the public key defines the public 
transformation. 
5 " Symmetric key " : A cryptographic key used 

with a symmetric cryptographic technique and known only 
to a set of specified entities. 

" Public Key Infrastructure (PKI) " : An organi- 
zation (or processes which acts on behalf of it) which 
10 offers services for the generation, registration, certi- 
fication, distribution, validation, and revocation of a 
certificate associated with an asymmetric key. pair. 

" Publ ic watermark " : A watermark that can be 
detected using a publicly available key (or a hash value 
15 thereof) . 

" Private watermark " : A watermark that can 
only be detected using a secret key (or a hash value 
thereof) and some data associated to specific cover data. 
It is not possible for an unauthorized third party to 
20 overwrite or delete the private watermark without the 
cryptographic secret keying information. 

" Detection watermark " : A watermark that can 
only be detected using a secret key (or a hash value 
thereof) . It is not possible for an unauthorized third 
25 party to overwrite or delete the private watermark with- 
out the cryptographic secret keying information. 

" Pay load " : The core of the hidden IAD in bit 
form without error control coding applied. 

" Image ID " : The following format scheme for a 
30 globally unique ID: The first 3 bytes determine the CCC , 
the following 3 bytes determine the CH ID defined by the 
CCC. Finally the CH can freely assign last 4 bytes for 
each one of his digital images or videos . 

" Oblivious " : A watermarking technique which 
35 does not require the cover-image for extracting the mark. 
In other words, only the stego-image is required to ex- 
tract the mark when using sin oblivious marking scheme. 
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" Template " : A hidden message encoded in the 
image.. Two kind of templates are used: "RST template (Ro- 
tation-Scale Template) " and "PST template" (Proportion- 
Scale Template) . By detecting the RST template, the scal- 
5 ing (zooming) and rotation suffered by a stego-image can 
be determined. By detecting the PST template, the hori- 
zontal and the vertical scaling are detected, and there- 
fore the change of proportion suffered by a stego-image 
can be determined. 
10 " Pseudo random seed tf : A value used to ini- 

tialize a pseudo random number generator. 

" Modulation" : Changing a component ' s value 
e.g. by addition or multiplication. 

15 Symbols: 

H, C, B, I 

Distinguished (unique) name of the Copyright 
20 Holder, the Copyright Certificate Center, the Buyer 

B and the Public Key Infrastructure I. 
Cert H , Cert C, Cert B 

Entity H's public key certificate from I, entity 
C's public key certificate from I and entity B's 
25 public key certificate from I. 

(ps x ,vs x ) 

The asymmetric signature and verification key pair 
of an entity with the distinguished name X. 
(pc x ,vc x ) 

30 The asymmetric decipherment and encipherment key 

pair of an entity with the distinguished name X. 

CC 

A copyright certificate 
DSSMRo (X,Y,Z) 

35 A digital signature generation scheme with message 

recovery, where X denotes the private key, Y the 
input data, and Z the resulting signature. 
DSSMRv (X, Y, Z ) 
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A signature verification scheme with message recov- 
ery, where X denotes the public key, Y the input 
data, and Z the resulting output data. 
DSSAP G (X, Y,Z) 

5 A digital signature generation scheme with appen- 

dix, where X denotes the private key, Y the input 
data, and Z the resulting signature. 
DSSAPy (X, Y, Z) 

A signature verification scheme with appendix, 
10 where X denotes the public key, Y the input data, 

and Z the resulting output data, 
crh A collision resistant hash function 
0WEA(X,Y,CD, SD) 

The oblivious, spread spectrum based watermark em- 
15 bedding algorithm with the seed X, the payload Y, 

the cover data CD, and the resulting stego data SD. 
0WVA(X, SD, Y) 

The oblivious, spread spectrum based watermark 
verification algorithm with the seed X, the stego 
20 data SD, and the resulting payload Y) . 

TVP 

Time variant parameter, such as a sequence number 
or a time stamp. 
RPMG(X,Y) 

25 A random phase mask generator, where X denotes the 

cryptographic key as input data and Y denotes the 
resulting phase mask as output data. 
DIES (PM, OI, CD) 

A symmetric digital image encryption scheme, which 

30 is based on the Fourier transform of the image, 

phase modification (random mask encoding by multi- 
plication on the complex exponential component 
e Mm.n) ^ ^ inverse Fourier transform, and quantiza- 
tion, where PM denotes the phase mask and ID de- 

35 notes the original image as input data and OI de- 

notes the ciphered image as output data. 
FFTS (CO, S R . SMC) 
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A component selector function of the real and 
imaginary FFT components . CO denotes the cover im- 
age, S R the applied selection rule function, and 
SMC the resulting set of FFT magnitude components. 

5 AF(SMC, HF, MS) 

An authentication function of the selected FFT mag- 
nitude components, where SMC denotes the identified 
magnitude components, HF denotes the applied crh, 
and MS the resulting authentication message as a 

0 string of arbitrary length. For example, AF(SMC, 

HF, MS) consists of generating a string from each 
selected Fourier component, concatenating, these 
strings and applying a hash function to the result- 
ing string . 

5 K X Y 

A secret key for a symmetric cryptosystem shared 
between two entities with the distinguished name X 
and Y . 
KxYt Data ] 

0 denotes the cipher text generated by a symmetric 

cryptosystem with plain text Data. 

ii 

Concatenation of two data elements. 

CD 

5 Cover Data 

SD 

Stego Data 

0 II. Copyright /Content /Originality protection based on a 
spread spectrum technique 

Depending on the proof-level to be provided 
for the protection, the preferred embodiment of the appa- 
5 ratus and method according to the invention provides 

three different levels of reliability, which are based on 
each other, namely: individual copyright /content /origin- 
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ali ty protection , copyright /content /originality protec- 
tion with registered cryptographic keys, and copy- 
right/content/originality protection with an CCC on the 
basis of registered cryptographic keys. 
5 Due to commercial requirements, the system 

provides different protection aspects, nameley content 
protection, copyright protection, and originality verifi- 
cation of the stego data. 

The copyright protection of a multimedia data 
10 set is considered as the process of proving the intellec- 
tual property rights to a court of law against unauthor- 
ized reproduction, processing, transformation, or broad- 
casting on the basis of digital evidence data. This proc- 
ess is based on a watermarking process WP and a registra- 
15 tion process RP . RP is executed after WP has been initi- 
ated and finished. RP is executed by a third party, which 
represents a different legal entity as the Copyright 
Holder (CH) , and provides digital evidence data for the 
CH required for verifying copyright ownership. The spe- 
20 cific cover- or stego data is a digital image, or video 
data. The WP embeds or extracts owner authentication data 
in or from multimedia data sets. This owner authentica- 
tion data is embedded such that the commercial usability 
of the multimedia data set is not affected. For this pur- 
25 pose, a key is applied to embed encoded owner authentica- 
tion data, called the watermark, into the cover data set 
I, resulting in a stego data set I*. The watermark data 
can then be extracted from the stego data if the correct 
key is used. 

30 In the following, WP is based on a perceptu- 

ally adaptive spread spectrum technique, a specific type 
of a symmetric cryptographic system. In order to embed or 
extract a watermark, it is necessary to know the exact 
values of the seed used for the generation of pseudo ran- 

35 dom sequences used to encode the watermark. Because 
spread spectrum signals are statistically independent 
(and therefore virtually orthogonal), more than one wa- 
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termark may be encoded into the multimedia data set. De- 
pending on the seed applied for the embedding and verifi- 
cation, we distinguish between a private and a detection 
watermark. A private watermark is defined as encoded 
5 owner authentication data embedded with a cryptographic 
signature as the seed. A detection watermark is defined 
as encoded owner authentication data embedded with a 
cryptographic secret key as the seed. We differentiate 
between copyright protection, content protection, and 

10 originality protection. 

Originality protection is considered as a 
process applied' after the copyright protection process. 
It enables a third party to check if the image contents 
has been modified on the basis of a public watermark. 

15 Content protection is considered as an addi- 

tional process applied during the trading transaction be- 
tween a service provider and a customer. The content pro- 
tection described is based on the transform domain of the 
image data and not on cryptographic ciphering algorithms 

20 applied during the communication between the service pro- 
vider and the customer, since these cryptographic algo- 
rithms are not robust against loosely compression and 
other image transformations. In addition, the performance 
of ciphering algorithms for the content protection of im- 

25 age or video data is very time consuming. 

The present method and apparatus is based on 
an image or video watermark technique, described below, 
which embeds and detects the the payload of a watermark. 
This technique is based on a perceptually adaptive spread 
spectrum technique which provides reliable means of em- 
bedding robust watermarks. Such a technique will be dis- 
cussed in section III. In addition, a spread spectrum 
techniques is a form of symmetric cryptosystem . In order 
to embed or extract a watermark, it is necessary to know 
the exact values of the seed used to produce pseudo ran- 
dom sequences used -to encode a watermark. The seeds are 
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considered to be cryptographic keys for watermark genera- 
tion and verification. System security can therefore be 
based on proprietary knowledge of the keys and provide in 
addition the necessary security parameters needed for a 
5 secure communication (mutual authentication, integrity, 
confidentiality, non-repudiation) in the trading process 
of digital images or videos. Because spread spectrum sig- 
nals are statistically independent (and therefore virtu- 
ally orthogonal), the present method and apparatus en- 

10 codes more than one watermark in an image or video frame 
at the same time, namely detection, private watermarks 
and public watermarks. The detection watermark allows to 
identify during a scanning process if the stego data be- 
longs to the copyright material of a CH. The generation 

15 of the private watermark is based on a digital signature 
as the seed and supports, therefore, third party verifi- 
cation who has generated the seed information for the 
coding and the decoding of the payload. The generation of 
the public watermarks enable the verification of the 

20 originality of the received stego data, on the private 
key of the asymmetric key pair of the ICH. 

Since the system provides for the registra- 
tion of the public key of the asymmetric key pair, the CH 
can prove that he is the only person in the possession of 

25 the adequate private key of the asymmetric key pair and, 
therefore, the generator of the private watermarks. 

The system also provides the secure registra- 
tion (mutual authentication, integrity, non-repudiation) 
of wacermark encoded images (stego data sets) at a CCC . 

30 The stego-image is registered at the CCC and a digital 

copyright certificate is generated which is signed by the 
CCC. If an unauthorized third party has also encoded wa- 
termarks in the same image, conflicting claims in copy- 
right disputes can be resolved. Examining the time stamps 

35 of the copyright certificate enables the secure identifi- 
cation of the legal owner: The earliest of the time 
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stamps identifies the legal owner if no copyright revoca- 
tion request has been applied. 

Watermark protection with registered crypto- 
graphic keys and the CCC based copyright protection are 
5 based on a PKI . The PKI issues on request public key cer- 
tificates containing the public key of the party, the 
distinguished name of the party, and a time stamp. Every 
certificate is signed with the PKI's private key and the 
trust is built on the validity of the authentic copy of 
10 the PKI's public key (we assume that the public key of 
the PKI is accessible, authentically distributed, and 
verifiable by every party) . 

In the following three levels of the system 
are described. 

15 

The method described in this section II re- 
quires a suitable watermarking technique. Various such 
techniques are known and can be employed. However, a pre- 
ferred technique is described in the section III. 

20 II. a) Registration based copyright, content, 

and originality protection 

Depending on the proof -level to be provided 
for the protection, our approach provides three different 
protection levels, which are based on each other, namely 

25 individual copyright /content /originality protection, 

copyright/content/originality protection with registered 
cryptographic keys, and copyright /content /originality 
protection with a CCC on the basis of registered crypto- 
graphic keys. Since the first two cases are special cases 

30 of the third one, we present only the approach for the 

registration based copyright protection. Depending on the 
level of protection to be provided (content or original- 
ity or copyright protection) , not all phases described 
below have to be executed. The phases described below 

3 5 have to be executed for the highest level of protection, 
i.e. content and originality and copyright protection. 
Based on one asymmetric key pair only, H can enforce the 
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different protection mechanisms for copyright, original- 
ity, and content protection. 

As shown in Fig. 3, the system for the CCC 
based protection is partitioned into four processes, 
5 namely the CH with the name H, the B process with the 

name B, the PKI process with the name I, and the CCC pro- 
cess with the name C. Suppose (ps H ,vs H ) , (pc H , vc H ) / (ps B 
,vs B ) , (pc B ,vc B ) , (ps T ,vs x ) , (pc x , vcj ) , (ps c ,vs c ) , and 
(pc c / vc c ) are the asymmetric key pairs of H, B, I and C, 

10 respectively and all. the involved parties would like to 
exchange information by on-line communication. (In the 
case of off-line communication, the security mechanisms 
to be provided for the communication are covered by op- 
erational means) . H has an authentic copy of Cert B and 

15 Cert c whose signatures were verified with the authentic 
copy of vs r . B has an authentic copy of Cert H and Cer*t c 
whose signatures were verified with the authentic copy of 
vsj. C has an authentic copy of Cert H and Cert B whose sig- 
natures were verified with the authentic copy of vs r . The 

20 following phases are then applied: 

Phase 1: 

H retrieves the cover data CD, generates a unique identi- 
fier ID CD := crh(H||SN), where SN is a serial number, 
stores ID CD / and retrieves the key pair (ps H , vs H ) . 
Phase 2 : 

Detection watermark embedding (image owner authentication 
and copyright protection) 

H generates the stego data SD applying the transforma- 
tion: OWEA(crh(ps H ) , SN||SN, CD, SD) . 
Phase 3 : 

Private watermark embedding (copyright protection) 

1. H generates the private Owner Authentication Data 
OAD CD applying DSSMRc(ps H , ID CD / OAD CD ) - 

2. H generates the stego data SD applying the transfor- 
mation: OWEA (crh (OAD CD ) / ID CD / CD, SD) , where CD is 
the SD of the last' phase. 



30 
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Phase 4: 

Public watermark embedding (originality protection) 

1. H generates the set of magnitude components, apply- 
ing FFTS (CD, S J MC) , with the selection function S 

5 and the resulting set MC of the FFT magnitude compo- 

nents. S is given by the normalization of the magni- 
tude components with the JPEG or MPEG quantization 
table entries and constrained by these components 
that will be modified during the coding process of 

10 the public watermark. 

2 . H then generates the authentication data for origi- 
nality verification, applying AF (MC , crh, AM), where 
MC denotes the in the last step generated FFT magni- 
tude component set, crh the applied hashing func- 

15 tion, and AM the resulting authentication message as 

output. AM is generated by converting the value of 
every magnitude component into a string and concate- 
nating the resulting strings of every magnitude com- 
ponent into one string. 

20 3 . AM is then ciphered with the key pc K , i.e. pc H [AM] 

and embedded as the payload in the public watermark, 
applying OWEA (crh ( vs H ) , pc H [AM] , CD, SD) , where CD is 
the SD of the last phase. 
Phase 5: 

25 H then stores the resulting stego data SD. 
Phase 6: 

H and C execute the following steps for the secure regis- 
tration or validation of copyright requests, and the gen- 
eration of copyright certificates. 
30 1. H generates first the copyright request data CRD, 
CRD := crh(SD||SN) and then the copyright request 
CR, CR := <TD| |SigTD>, with TD : = CRD | | TVP|||h||C, 
and DSSAP G (ps H/ TD, SigTD) . H then transmits CR to C. 

2. C receives CR and verifies TD, applying DSSAP V (vs H , 
35 SigTD, IVR) , where IVR denotes the intermediate 

verification result. If IVR = crh(TD), with TD := 
CRD | |TVP| |H| |C, then TD has been successfully veri- 
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fied and the next step shall be executed. In any 
other case, the processing and communication between 
the H and C is stopped. 

3. If verification was successful, C generates the cor- 
5 responding digital copyright certificate executing 

DSSAP G (ps c , CCD, SigCCD) , with CCD : = CRD | | TVP . C then 
stores the copyright certificate CC := CCD||SigCC 
and generates then the Copyright Confirmation Reply 
CCR, CCR := <TD| | SigTD> , with TD : = CC | | TVP | |C| |H, 
10 and DSSAP G (ps c , TD, SigTD) . C then transmits CCR to 

H . 

4. H receives CCR and verifies TD, applying DSSAP v (vs H , 
SigTD, IVR) , where IVR denotes the intermediate 
verification result. If IVR = crh(TDT, with TD := 

15 CC | | TVP | [C| |H, then TD has been successfully veri- 

fied. H then verifies and stores the CC . The follow- 
ing phase can now be executed repeatedly, if neces- 
sary, without repetition of the previous phases. 
Phase 7 : 

20 H and 3 execute the following steps for the trading of 
copyright, content, and originality protected digital 
data (images and video) : 

1. B generates the trading transaction Tl , Tl := 
<TD | | SigTD> , with TD := ID CD | | TVP | | B | | H , and 

25 DSSAP G (ps B , TD. SigTD) . B then transmits Tl to H. 

2. H receives Tl , verifies TD, applying DSSAP v (vs B , 
SigTD, IVR) where IVR denotes the intermediate veri- 
fication result. If IVR = crh(TD), with D.:= 

ID CD | I TVP I |B| |H, then TD has been successfully veri- 
30 fied and the next step shall be executed. In any 

other case, the processing and communication between 
the H and B is stopped. 

3. If the verification was successful, H retrieves with 
the ID CD information the corresponding stego data SD 

35 and generates the trading transaction T2 := 

<TD | | SigTD> , with TD := CD | | TVP | | H | | B , DIES(PM, SD, 
CD) with RPMG (DSSMRc (ps H , B||SN), PM) , and DSSAP G (ps H , 
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TD. SigTD) . B | | SN designates the B and the picture 
and is called the mask message. H then stores 
DSSMR G (ps H , B||SN) and transmits T2 to B. 
Phase 8 : 

5 B receives T2 and verifies TD, applying DSSAP v (vs H/ SigTD, 
IVR) , where IVR denotes the intermediate verification re- 
sult. If IVR = crh(TD) , with TD := CD | | TVP | | H | | B , then TD 
has been successfully verified and CD is locally stored. 
Phase 9 : 

10 After B has paid, H retrieves IK B and sends vc B [IK B ] . B 

receives vc B [IK B ], deciphers it (pc B .[ vc B [ IK B ] ] ) , and gener- 
ates the random phase mask PM. This random phase mask is 
then used for deciphering CD (DIES (PM, CD, SD) ) to get the 
original stego data SD. 

15 Phase 10: 

B may verify the originality of the stego data SD, re- 
trieving the public key from H and applying 

OWVA (crh ( vs H ) , SD, pc H [AM] ) . B then deciphers pc H [AM] apply- 
ing vc H [pc H [AM] ] . H then verifies AM applying the same 

20 steps 1 and 2 as described in phase 4. If the verifica- 
tion was successful, the image content has not been al- 
tered. If the watermark has been destroyed or overwrit- 
ten, the contents of the SD has been modified. If the 
verification fails, the content has also been modified by 

25 unauthorized parties. 

Remark : 

Depending on the applied asymmetric scheme 
the private decipherment key may be identical to the pri- 

3 0 vate signature key and the public enc ipherment key may be 
identical with the public verification key. 

Since the generated asymmetric key pairs are 
unique, the CH can be uniquely identified on the basis of 
the digital copyright certificate. 

35 B may check the copyright certificate re- 

questing C (or H) to transfer an authentic copy of the 
copyright certificate for a given identifier ID CD . Except 
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the data transferred, the applied protocol is the same as 
described above (see phase 6) . 

If H would like to transfer a specific copy- 
right of a CD set to another legal party, he may initiate 
5 a copyright revocation request with C. The different 

phases of this request are analogue to the copyright re- 
quest . 

For copyright verification, the CH first 
verifies the detection watermark and then the private wa- 
10 termark with the extracted SN. 

Copyright verification may be checked by a 
third party, if the H transfers the digital signature ap- 
plied for the seed. Based on the retrived bublic key from 
H, .the third party can verify that H is the only one who 
15 has generated the corresponding signature. 

II. b) Copyright, content, and originality 
protection with registered keys 

As shown in Fig. 2, the apparatus for the 

20 copyright, content, and originality protection with reg- 
istered cryptographic keys is partitioned into three pro- 
cesses, namely the CH with the name H, the Buyer process 
with the name B, and the PKI process with the name I. 
Suppose (ps H ,vs H ), (pc H /Vc H ), (ps B , vs B ) , (pc B , vc B ) , 

25 (psj, vsj) , and (pcj, vcj) are asymmetric key pairs of H, 
B, and I, respectively. Suppose H has an authentic and 
actual copy of Cert B which signature was verified with 
the authentic copy of vsj and the B has an authentic and 
actual copy of Certpj which signature was verified with 

30 the authentic copy of vsj . Then the same phases except 
phase 6 as for I I. a) have to be applied. 
Remark : 

Since the generated asymmetric key pairs are 
unique, the CH can be uniquely identified if no addi- 
3 5 tional watermarks by unauthorized persons have been en- 
coded into the SD. 
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II.c) Individual copyright, content, and 
originality protection 

As shown in Fig. 1, the apparatus for the in- 
5 dividual copyright , content, and originality protection 
is partitioned into two processes, namely the CH with the 
distinguished name H and the B process with the distin- 
guished name B. Suppose (ps^. vs^) and (pc H , vc H ) are 
asymmetric key pairs of H, (psg, vsg) and (pc B , vcg) are 
10 the asymmetric key pairs of B. Suppose H has an authentic 
copy of vsg, vc B and B has an authentic copy of vs^/ vch- 
Then the same phases as for II. b) have to be applied. 

Remark : 

In the case of a legal copyright dispute, H 
15 can retrieve the payload of the detection watermark and 
construct the signature taken as the seed for the private 
watermark. Since the generation of the same asymmetric 
key pair by two distinguished entities is very unlikely, 
the generation of the digital signature as the seed for 
20 the private watermark provides a good level of proof 

against copyright infringement. In the case of watermark 
protection with registered keys, the generation of the 
same asymmetric key pair by two distinguished entities 
can be prevented. 

25 

III. Embedding the watermarks 

The watermarking technique described here 
30 comprises the following steps : 

a) An error-control coding technique for the 
message to be transmitted in the watermark; 

b) A method to encode respectively to decode 
the message resulting from step a) ; 

35 c) A reliable method for embedding the en- 

coded message from step b) in the image or video without 
introducing visible artifacts. 
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d) A watermark extraction technique that is 
robust against compression, translation, rotation, scal- 
ing or change of proportion of the stego image or video. 

e) A watermarking technique for small and or 
irregular blocks . 

f) A method that allows to detect if a stego- 
image was marked or not with a given key without extract- 
ing the encoded message. 

g) A method for watermarking without template 
which is resistant to translation, rotation and scaling. 

h) A method for watermarking videos. 

Each of these aspects can be applied to con- 
ventional watermarking techniques. Preferably, chey are 
used in combination to provide a highly reliable, robust 
and powerful method for marking data sets. This method 
can be applied for any watermarking applications, in par- 
ticular to the application described in section II of 
this disclosure. 

Steps a) and b) can be used for embedding wa- 
termarks in any type of data while steps c) is optimized 
for embedding watermarks in images or video frames . 

In the following, the above mentioned ele- 
ments of the watermarking technique are described in de- 
tail. 

III. a) Error control coding 

Error control coding is applied to the mes- 
sage prior to encoding seep Ill.b). When used in combina- 
tion with the procedure described in section II, the mes- 
sage corresponds to one of the blocks BL^ . 

Preferably, symbol based Reed Solomon (RS) 
codes are applied for this purpose. The advantages are 
the following: 

- RS codes correct symbol errors rather than 
bit errors, and 
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- RS codes can correct erasures as well as 
errors. Erasures can be factored out of the key equation, 
which means that "erased" symbols can be ignored. They do 
not play any role in the error control mechanism - an 

5 erasure is useless redundancy. 

Being able to discard erased symbols has two 

advantages : 

- If the posterior probability of a received 
symbol is low, it may be ignored. 

!0 - RS codes only come in standard sizes. For 

example a 255 x 8 bit code is common. Most commonly used 
RS error control codes appear to be too large to be used 
in watermarking. However, it is possible to make almost 
any RS code fit a watermarking application by judiciously 

15 selecting symbols as being erased (because they were 
never embedded in the image in the first place) . 

Ill.b) Encoding the message 

20 During encoding, the message to be transmit- 

ted in the watermark is transformed into a form suited 
for being used in the modulation of image components . At 
the same time, it is encrypted using a suitable key. 

If used with the method of section II, the 

25 encoding procedure has access to the cryptographic keys 
p H and vh (or their hash values), which are applied as 
seeds to generate pseudo-random sequences as described 
below. The public key is used for encoding the message of 
the public watermark, the private key is used for the 

30 private watermark. Knowledge of the corresponding key (or 
hash value) is required for recovering the message from 
the watermark. 

A watermark may be embedded or extracted by 
the key owner. In this form spread spectrum is a symmet- 

35 ric key cryptosystem. From the point of view of embedding 
watermarks in images or videos given the cryptographic 
keys the sequences themselves can be generated. A good 
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spread spectrum sequence is one which combines desirable 
statistical properties such as uniformly low cross corre- 
lation with cryptographic security. 

Suppose we are given a message B (e.g. that 
5 was provided with error coding in above step III. a) . The 
message has the binary form bib2-..b L/ where bi are its 
bits. This can be written in the form of a set of symbols 
S1S2 • . < s M - most generally by a change in a number 
base from 2 to B. The next stage is to encode each symbol 

10 si in the form of a pseudo random vector of length N, 
wherein each element of this vector either takes the 
value 0 or 1. N is e.g. in the order of 1000 to 20000 (in 
the order of 10%-50% of the total number of image coeffi- 
cients (Fourier components) that can, theoretically, be 

is modulated) . 

In a preferred embodiment, this is carried 
out by using a pseudo random generator seeded by the key 
crh(PH) or crh(v H ) . 

To encode the first symbol a pseudo random 

20 sequence v of length N + B - 1 is generated. To encode a 
symbol of values where 0 < s < B the elements v s , v s + i 
* • ■ V s+N-1 are extracted as a vector of length N. For 
the next symbol another independent pseudo random se- 
quence is generated and the symbol encoded as a random 
• 25 vector T2 • Eac ^ successive symbol is encoded in the same 
way. Note that even if the same symbol occurs in differ- 
ent positions in the sequence, no collision is possible 
because the random sequences used to encode them are dif- 
ferent - in fact they are statistically independent. Fi- 

30 nally the entire sequence of symbols is encoded as the 
summation : 

m =£i=i . . m r j_ 

The pseudo-random vector m has N elements, 
each varying between 0 and M. In a next step, the ele- 
35 ments of m are offset to make their mean zero. These ele- 
ments will determine the strength of modulation of the 
Fourier components of the image in step III.c. 
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When decoding the watermark, a vector m' 
(read-out message) is derived from the stego-image. In 
oblivious watermarking, m' corresponds to the modulated 
Fourier coefficients. Hence, in general m' will not be 
5 equal but "similar", to m. 

To decode s from m' , the elements of m' are 
first offset to make their mean zero. Then, starting from 
the (known) seed, the first random sequence v of length N 
+ B - 1 is generated and the correlation of v with m' is 
o calculated. The peak of the correlation indicates the 

offset sj_ in the random sequence that was used for gener- 
ating rj_ b Then, the next random sequence v is generated 
and cross-correlated with m' to retrieve S2 , etc. 

Reliable communications of the apparatus are 
5 best accommodated by using m-sequences or Gold Codes to 
generate the random sequences r-j_ and use amplitude modu- 
lation: 

where bi and are bi and in which each bit 0 was 

replaced by 1 and each bit 1 by -1 due to the isomorphism 
between the group (exclusive OR, {0,1}) and (*,{1,-1}). In 
this case the values of m are between -M and M. Then the 
decoding is carried out by simply cross correlating with 
each of the random sequences r-j_ in turn. If the correla- 
tion is negative then a binary one has been sent, other- 
wise a binary 0 . 

Gold codes and m-sequences, both insure a 
good reliability and security of the embedded mark. How- 
ever, Gold codes have the advantage that for a given 
register length k (N=2 k -1) there is a larger choice for 
the key (2 2Jc -l instead of 2 k -l) and a better correlation 
properties if only part of the sequence is used. 
If M is sufficiently large, the statistical distribution 
of the message m should approach a Gaussian (Central 
Limit Theorem) . A Gaussian distributed watermark has the 
advantage that it is more difficult to detect. The vari- 
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ance increases with order M 1/2 ; in other words, the ex- 
pected peak excursion of the sequence is only order M i/2 . 

III.c) Embedding the message in the image or video 

In this step, the encoded message m (e.g. as 
obtained in the previous step) is applied to the image or 
a video for generating the watermark. 

In contrast to steps III. a) and Ill.b), em- 
bedding the message in the image requires some knowledge 
of the nature of the data stored in the image. In the 
following, the image is assumed to be a two-dimensional 
image that can be a still image or a video frame. The 
method is optimized for robustness against operations 
generally applied to images or video frames such as 
translation,, cropping, rotating, scaling, change of pro- 
portion. (The method is not optimized for other types of 
data, such as sound or text.) 

In order to achieve robustness against circu- 
lar translation, the image block is first: subjected to a 
Fourier transform. Then, message m is used to modulate 
the Fourier components. In addition to this, a template 
is embedded in the image, which template can be used for 
detecting rotation, scaling or change of proportion of 
the image when reading the watermark. A tiling mechanism 
and suitable phase-dependent correction are applied for 
providing robustness against cropping. 

Figure 4 shows a detailed diagram describing 
the embedding of the watermark. Calculation starts from 
the cover image: 

1. If the image is a color image, then compute the lumi- 
nance component (by replacing each pixel by g/2 + r/3 
+ b/6, where g, r and b are its green, red and blue 
components) and use these values for the following 
calculations . 

2. If a predefined block size (N b ) is used, divide the im- 
age into adjacent blocks of size N b x N b (e.g. 128 x 
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12 8 pixels) . Otherwise N b is the minimum of the image 
height and width <N b =min (height, width) ) . 

3. Map the image luminance levels (or gray levels for a 
black and white image) because it corresponds to a 
perceptually "flat" domain by replacing them with 
their logarithm. The logarithm is a good choice be- 
cause it corresponds of the Weber-Fechner law which 
describes the response of the human visual system to 
changes of luminance. 

4. Compute the FFT (Fast Fourier Transform) of each 
block. From the real and imaginary components obtained 
in this way, calculate corresponding magnitude and 
phase components . 

The magnitude components are translation invariant and 
will therefore be used in the following modulation 
steps. (However, it is possible to derive translation 
invariants from the phase spectrum as well, which 
could also be modulated) . 

5. Select the magnitude components to be modulated. To 
encode a message m of length N, a total number of N 
components are modulated. In non-oblivious watermark- 
ing, any components can be modulated. For oblivious 
watermarking, because of interference of the cover im- 
age with the watermark, the largest (highest energy) 
components (at about the lowest 10% of the frequen- 
cies) are avoided and components at medium frequencies 
(about next 30%-50%) are used; these frequencies are 
adjacent and are thus located in a band of frequen- 
cies. These figures are chosen because they generally 
give a good compromise between robustness and visibil- 
ity of the watermark. 

There are several methods for selecting the components 
to be modulated, for example: 

a) The selection of the components to be modulated 
does not depend on the given image. Rather, the 
same components are selected for every image. The 
author as well as the reader of the watermark know 
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either the positions of the components to be se- 
lected in advance or a key which allows by means 
of a pseudo-random generator seeded by this key to 
generate the positions. 
5 b) The largest, components (inside the allowable fre- 

quency range) are used for modulation, 
c) Almost all magnitude components in a given fre- 
quency band are used. The upper limit of the band 
is computed such that the number of frequencies 
10 inside the band be larger than and as close as 

possible to N. 

■ in the methods b) and c) the order in which the compo- 
nents to be modulated can be provided by a pseudo- 
random generator seeded by a key known by both, author 

15 and reader. 

When selecting the components to be modulated, care 
must be taken to preserve the symmetry imposed on the 
Fourier components F(ki, k2 ) by the fact that the im- 
age block is real valued: 

20 F(ki, k 2 ) = FMN b - ki, N b - k 2 ) 

Once the magnitude components (M 1# ... M N ) to be modu- 
lated are chosen, the corresponding value m^ of mes- 
sage xn is added to or subtracted from the correspond- 
ing selected magnitude component Mi. Addition is used, 

25 if the corresponding phase component Pi is between 0 

and 71, subtraction if it is between k and 2n. This 
provides robustness against translation and cropping 
(see below) . 

Before adding/subtracting the values mi to/ from Mi, 
30 the vector m can be scaled to adjust the magnitude of 

its elements to those of the components Mi- 
Generally, the elements mi should be of the same order 
of magnitude as the components Mi. The depth of modu- 
lation or amplitude of the embedded signal should de- 
35 pend on the objective measure of the perceptual sig- 

nificance. The lower the perceptual significance, the 
higher should be the amplitude of the watermark. 



WO 99/17536 



PCT/IB98/01500 



36 

Moreover, to insure a good invisibility one can use 
local energy and masking criterion (see J . F . Delaigle, 
C. De Vleeschouwer, B. Macq, "Digital watermarking", 
Proceedings of the SPIE Electronic Imaging: Science 
5 and Technology, vol. 2659: Optical Security and coun- 

terfeit Deterrence Techniques, San Jose, February 
1996) to determine the depth of modulation. However, 
for simplicity, the amplitude for all components is 
kept constant. This constant can be predefined by the 

10 owner or can be some function of the mean and/or the 

variance of the energy in the image or its Fourier 
transform and the values of the pseudo-random vector m 
containing the encoded message (e.g. (mean ( energy ) + a 
* variance (energy) ) /mean (m) , where a is a predefined 

15 constant) . 

6. Add a template by a second modulation of the magnitude 
components. This is described in more detail below. 

7 . Compute the inverse FFT using the phase components and 
the modulated magnitude components . 

20 8 . Compute the inverse of the perceptual mapping function 
of step 3. For Weber-Fechner law mapping, the inverse 
function is an exponential. 
9 . Replace each watermarked block in the image to obtain 
the scego- image. 

25 10. If the image is a color image, then rescale the red, 
green and blue components by the relative change in 
luminance introduced by embedding a watermark. Typi- 
cally, the red, green and blue pixels occupy a byte 
each in program memory. If overflow or underflow oc- 

30 curs then the pixel is set to the upper bound 255 or 

lower bound 0 respectively. 

Template : 

As mentioned above, a template is added to the image in 
35 step 6. Two kinds of templates can be used: 

a) a RST template - to detect rotations and scaling 
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b) a PST template - to detect horizontal and verti- 
cal scaling. 

The PST template is rather used in case of video frames 
(changes of proportion are more likely to occur in the 
case of videos than rotations) and the RST is rather used 
for still images (photographs, paintings, etc,...). 
The steps for generating the template are illustrated in 
Fig . 5 : 

20. Apply a log-polar or a log-log map to the magnitude 
components. The log-polar map transforms the magni- 
tude components of the FFT into a polar coordinate 
system (0, log-r) with logarithmic radius axis as 
follows. Consider a point (x,y)e9v 2 and define: 

x- e M cos 0 

y = e M sin © 

where |ie<K and 0 < 0 < 2k. If r= e M , |l= log(r) and 
for every point(x,y) there is a unique (0,log(r)) 
that corresponds to it. In the log-polar representa- 
tion, a scaling of the image leads to an offset of 
the components along the log-r axis and a rotation of 
the image leads to an offset along the 0 axis. Simi- 
larly, the log- log map transforms the magnitude com- 
ponents into a logarithmic coordinate system (log-x, 
log-y) as follows. For each point (x,y)e^ 2 define: 




Then, ot=log(x) and p=log(y), and in this log-log rep- 
resentation, the horizontal respectively vertical 
scaling leads to offsets along the log-x respectively 
log-y axes . 

21. Preferably, low pass filtering is used for interpo- 
lating the frequency space components during this 
mapping. The magnitude components belonging to very 
low or high frequencies are not mapped. The following 
modulation is only applied to components in medium 
frequency range . 
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22 . Select the magnitude components in the log-polar or 
log-log coordinate system to be modulated. Typically, 
about 0.1-0.3% of all components are to be modu- 
lated. The RST or PST pattern T formed by the se- 
lected components in log-polar or log-log space 
should be such that its auto-correlation under trans- 
lation is weak. For this purpose, the indices of the 
selected components should be coprime or be derived 
from a two-dimensional random sequence. This random 
sequence can be generated by a random generator 
seeded by a key K. Whoever knows this key K will be 
able to reconstruct the template and detect the wa- 
termark as explained below. Each selected component 
is increased by a given value. 

23 . Map the modulated points by change of coordinates 
back into frequency space ( inverse log-polar mapping 
or inverse log-log mapping) . 

The RST or PST pattern T formed by the selected compo- 
nents in log-polar respectively log-log space is prede- 
fined and known to the reader of the watermark. 
It must be noted that the calculation of the log-polar 
respectively log-log transform of the cover image or 
video frame is not required for generating the template. 
Instead, the RST or PST pattern T of the components to be 
modulated in log-polar respectively log-log space can be 
mapped back to frequency space, which results in a RST or 
PST pattern T' in frequency space that can be applied di- 
rectly to (e.g. added to) the components in frequency 
space. Alternatively, the template can be added directly 
in the Fourier transform domain. 

As will be explained below, the template is not required 
for non-oblivious watermarking. 

Ill.d) Extracting the watermark from the stego-image or 
video 
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Figure 6 shows a detailed diagram illustrating the steps 
for reading a watermark from the stego-image or stego 
video frame: 

31. If the image is a color image then compute the lumi- 
5 nance component and use these values for the follow- 
ing calculations. 

32. If predefined block size (N b ) is used, divide the 
image into adjacent blocks of size N b x N b (e.g. 128 x 
128 pixels) . Otherwise N b = min (height, width) . 

10 33. Map the image luminance levels (or gray levels) to 
the perceptually "flat" domain by replacing them 
with their logarithm. 

34. For each block compute the FFT . 

35. Use a data windowing process to suppress the edge 
15 effects in the magnitude spectrum due to possible 

rotation or scaling of the image. Different windows 
can be used such as Blackman, Hamming, Hanning, 
Welch or Bartlett Window (see W.H. Press, S.A. Teu- 
kolsky, W.T. Vetterling, and B.P. Flannery, "Numeri- 

20 cal Recipes in C", Cambridge University Press, sec- 

ond edition, 1992). The effect of data windowing in 
the space domain is equivalent to convolution in the 
frequency domain with a narrow filter. The blurring 
effect introduced by this convolution is beneficial 

25 because it tends to smooth the spectrum which makes 

interpolation more effective. 

36. Determine the rotation and scaling that the image 
suffered by finding the RST template in log-polar 
space or determine the horizontal and vertical scal- 

30 ing by finding the PST template in the log-log 

space. These steps are described below in "Finding 
the template" section. 

37. Using the results of step 35, read the modulated 
components to generate message m' . This requires the 

3 5 knowledge of the method that was used in step 5 for 

selecting the components to be modulated. 
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Once that the message m' is recovered, it is demodulated 
and error corrected using the methods described in sec- 
tions III. a) and Ill.b). 



5 Finding the template : 

The steps for finding the template are illustrated in 
Fig. 7: 

40. Apply log-polar or log-log mapping to the magnitude 
components of the Fourier transform. The magnitude 

10 components belonging to very low or high frequencies 

are not mapped. The following analysis is only ap- 
plied to components in medium frequency range or to 
all components except the low frequency range. 

41. For oblivious watermarking, calculate the normalized 
15 cross correlation of the components in log-polar or 

log-log space with the RST or PST pattern T that was 
used for generating the template in step 21 and find 
the point of best correlation. If the image has nei- 
ther been rotated or scaled, this point is at zero. 

20 rf the image is rotated and/or globally scaled there 

is an offset along the 0 axis and/or log-r axis , in 
the log-polar map. If the scaling suffered by the 
image or video frame was different on horizontal 
respectively vertical axis, there are offsets along 

25 log-x respectively log-y axes in the log-log map. 

For non-oblivious watermarking, the log-polar re- 
spectively log-log transform of the Fourier compo- 
nents of the cover image can be used instead of RST 
or PST pattern T for retrieving scaling, rotation 

30 respectively change of proportion 

The cross correlation can be calculated efficiently 
using conventional Fourier techniques. 

In order to obtain better results and lower 
computational cost, before applying the cross correlation 

35 one can first adaptively filter the data to remove out- 
liers and noise and use a filter which keeps only local 
peaks- This can e.g. be carried out by locally calculat- 
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ing the variance (or some other value indicative of the 
data's distribution) of neighbouring data of each data 
point. If a given data point lies clearly outside this 
variance, is it replaced by zero. In a next step, local 
5 peaks that have not been filtered out are then stored in 
a sparse matrix to reduce computation. The fast correla- 
tion (using the FFT or by a point by point correlation) 
is done in this case between the peaks of (T) and the 
peaks of ( T ' ) • The correlation can moreover be weighted 

10 so that the more reliable central points are more 
strongly weighted. 

It is possible to further increase accuracy 
of the scaling and rotation factors by carrying out the 
following: detecting a scaling and/or rotation in a first 

15 iteration from the correlation between the log-polar or 
log-log transform and the template, using said scaling 
and/or rotation for -either a) scaling and/or or rotating 
said Fourier transform, calculating a scaled and/or ro- 
tated log-log or log-polar transform therefrom and corre- 

20 lating said rotated log-log or log-polar transform with 
said template, or b) calculating a second template by 
scaling and/or rotating an original Fourier-space tem- 
plate and calculating a log-log and or log-polar trans- 
form therefrom and using said second template for calcu- 

25 lation a second correlation with said log-log or log- 
polar transform of said stego data 

IlI.e) Embedding watermarks in small and/or irregular 
blocks 

30 

To embed watermark in small blocks,, one computes the 
transform over regions that instead of comprising only 
one block, extend over adjacent blocks. To do this one 
can use the Lapped Orthogonal Transform (see H.S. Malvar, 
35 "Signal Processing with Lapped Transforms", Norwood, MA, 
1991) which has the advantage to minimize blocking ef- 
fects which would otherwise make a strong watermark based 
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on blocks visible, especially for small block sizes. This 
is followed by the method as described in III.c and 
Ill.d, where the Fourier transformation phase is replaced 
by the Lapped Orthogonal Transform (LOT) application for 
5 the cover image, while keeping the same template opera- 
tions. 

Using small blocks (of roughly 16 by 16 points) allows 
the strength of the embedded message to be modulated as a 
function of the local variance, which renders the method 

0 adaptive. Furthermore the watermark can be recovered lo- 
cally the only requirement being that a sufficient number 
of blocks are available to contain 1 complete message. 
To embed watermark in blocks with irregular shapes 
(non-square and non-rectangular) such as might occur in 

5 MPEG4 video compression, two possible solutions can be 
applied: 

• padding of the irregular blocks in order to obtain 
square blocks, using either constant padding, or sym- 
metrical padding, then method as in III.c and III. d; 

0 • avoid the padding phase by directly using wavelet 

transforms of arbitrary length signals (see H.S. Bar- 
nard, Image and Video Coding Using wavelet decomposi- 
tion, CIP-Gegevens , Koninklijke Bibliotheek, Den Haag, 
1994). This is followed by the method as described in 

5 III.c and Ill.d, where the Fourier transformation 

phase is replaced by the Wavelet Transformation for 
the cover image, while keeping the same template op- 
erations . 

o Ill.f) Watermark detection without extraction 

Being able to detect a watermark without being 
able to decode it is useful and in many cases sufficient 
to prove the identity of the generator of the watermark. 
5 This can be done by a Bayesian approach (see J.J.K. 6 Ru- 
anaidh and W.J. Fitzgerald, "Numerical Bayesian Methods 
Applied to Signal Processing" , Series on Statistics and 
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Computing, Springer-Verlag , 1996) that allows to compute 
the probability that a watermark generated by a given key 
is present in the stego-image, relatively to the prob- 
ability that no watermark was generated with that key. 

The implementation of this principle operates 
as follows. The used watermark d is a linear combination 
of pseudo-random sequences corrupted by noise: 



d = G b + e 



10 



where e is a noise vector corrupting the watermark, b is 
an M- x 1 vector and G is an N x M matrix of bits in form 
+1 and -1 (due to the isomorphism between the group (ex- 
clusive OR, {0,1}) and (*,{1,-1>) 0 was changed to 1 and 1 
15 to -1) . Each column of G.is a pseudo-random sequence such 
as an m-sequences or a Gold Code in which 0 was changed 

to 1 and 1 to -1. 

If we assume that the noise follows a Gaussian 
distribution, the probability that a message of length M 
20 was embedded with a said key .k in the stego-image (SD) 

is : 



25 



30 



-1/2 



, „ x 7r" N/2 r(M / 2)H( N - M) / 2) det(G T G) 
M I d. SI) ~ ^,, (d T d „ f T f)( N- M v: 



where T is the gamma function, R 5 and R c are irrelevant 
constants introduced as normalization factors, 

b = (G T G) 1 G T d 

and 

f = G T b 

The probability that no message was embedded with the 
said key k in the stego-image (SD) is: 



P (/c,oid,s/)oc ; ) 



;r- N/2 r(N/ 2) 
2R a (d T d)' 
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Finally, we compute the relative log-probability: 

pik.Mid.sn 

B p(* f Old f 5/) 
and compare with 0 . 

5 Ill.g) Watermarking without template 

Using a combination of Fourier transform and a log- 
polar map, i.e. the Fourier-Mellin transform that is the 
Fourier transform of a log-polar map, allows to embed a 
o watermark in a domain that is invariant to rotation, 

scale and translation, without the need to use a template 
to detect rotations and scaling. The method consists of 
directly transforming the cover-image or -video frame in 
the log-polar domain; the watermark is directly inserted 
at this stage. Figure 8. shows the steps for embedding 
the watermark in a rotation, scale and translation in- 
variant domain. 

An alternative which is computationally more efficient 
bypasses the mapping of the original image or video frame 
in the rotation, scale and translation invariant domain. 
This is shown in Figure 9 . The scheme to extract the wa- 
termark from the image is shown in Figure 10. 

Replacing the log-polar mapping by the log-log map- 
ping allows to embed a watermark in a domain that is in- 
variant to translation, horizontal and vertical scaling. 

This is an idealized watermarking scheme which works 
in principle but which in practice is quite costly and 
difficult to implement. The first difficulty is that both 
the log-polar mapping (LPM) and the inverse log-polar 
mapping (ILPM) can cause a loss of image quality. The 
change of coordinate system means that some form of in- 
terpolation must be used. This leads to a second diffi- 
culty, which is rather numerical. Interpolation only per- 
forms well if the neighboring samples are of the same 
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scale, which is not verified by the magnitudes of the 
frequency components . 

II I. h) Watermarking videos 

5 

In the case of uncompressed video each frame 
is marked. One possibility is to use the same key and the 
same watermark in each frame. However this can decrease 
the robustness of the watermark against forgery. There- 
io fore, it is preferable to use the same key, but a differ- 
ent watermark for each frame. (e.g. the label of the video 
followed by the frame number) . In the case of MPEGl or 
MPEG2 compressed video, only the intraframes I (the first 
frame of each group of pictures) are marked. 
15 Another novel alternative for watermarking 

uncompressed video is to individually mark three- 
dimensional spatio temporal blocks of video stream, which 
may be overlapped in time and/or in space. The method 
used here is an extension of the algorithms used for 2D 
20 images to the temporal dimension, using 3D Fourier 

transform, 3D template, and the same spread spectrum 
techniques to generate the watermark. The use of Fourier 
transform ensures the same rotation, scaling, and pro- 
portion invariances. We have also a full invariant 3D wa- 
25 termark for theses blocks, exactly as for 2D still image 
watermarking. These 3D blocks may be rather large, or 
small enough to ensure more robustness against cropping. 
As for individual frame marking, we can use the same wa- 
termark for all blocks, or a different watermark for 
30 each block. The advantage of this spatio temporal ap- 
proach is to take in account the motion and scene varia- 
tion in watermarking, as developped in the paper of M.D. 
Swanson, B. Zhu and A.H. Tewfik, "Multiresolution Scene- 
Based Video Watermarking using Perceptual Models", IEEE 
35 Journal on Selected Areas in Communications, vol. 16, no. 
4, May 1998. However, in contrast with our apparatus, 
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they make use of ID temporal wavelets transform instead 
of our 3D Fourier transform. 

IV. Properties of the watermark: 

In the following, some of the properties of 
the watermark generated using the steps described above 
are discussed. 

Resistance to cropping: 

One feature of translation invariants devel- 
oped using the Fourier transform is that they are invari- 
ant to circular translations (or cyclic shifts) . This is 
used to construct watermarks that are invariant to crop- 
ping. This is illustrated by reference to Figs. 11 and 
12 . 

As mentioned above, the image is split into 
blocks and the watermark is applied to each block. In 
other words, the same modulation pattern is applied to 
the Fourier components of each block, wherein the modula- 
tion pattern is given by the corresponding encoded mes- 
sages m. 

Fig. 11 shows such an image where the fat 
lines 100 designate the borders between the blocks. Sup- 
pose that the watermark in a standard size block will be 
of the form; . 

T= [A B ; CD] 

where the sub-matrices A, B, C and D are of 
arbitrary size. A circular translation of such a water- 
mark is of the form: 

' S= CD C ; B A] . 

The original stego-image is tiled with water- 
marks in the pattern [T T T T ; T T T T ;T T T T] . There- 
fore, a cropped section of the matrix will carry a water- 
mark in the form [SSSS ; S S S S ; S S S S] . This is 
illustrated in Figure 12. When reading the watermark of 
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the cropped image of Fig. 12, each block carries the wa- 
termark S. Since S is a circular transform of T, it can 
be read without problems in the Fourier domain using the 
steps outlined above. 

5 Note, however, that the cover image is not 

tiled, only the watermark is. Therefore, while cropping 
merely induces a circular translation of the watermark in 
each block, the change of image in each block is not a 
circular translation. To compensate for this, the phase 

o components Pi of the Fourier transform must be used for 
correcting the sign of the modulation of the magnitude 
components Mi, as it is outlined under step 5 above. 

The optimum size of block depends on a number 
of different factors. A size that is a power of two is 

5 useful because the FFT can be used. The block size also 
must be small enough to withstand cropping but large 
enough to comfortably contain a watermark. The best com- 
promise for block size is 128. 

0 Resistance to scaling and rotation: 

As mentioned above, reading the RST template 
in log-polar space allows to detect and measure any scal- 
ing and/or rotation that was applied to the image. This 
information can then be used for reading the watermark. 

5 Since the reader knows the pattern that was used for 
modulating the magnitude components in step 5, he can 
identify the modulated components in the scaled and ro- 
tated image and derive the message m' therefrom. An al- 
ternative is to compensate the transformation using the 

o measured rotation and "scaling and read the message in the 
compensated image. 

Note that the apparatus does not explicitly 
use a rotation and scale invariant watermark but instead 
searches the parameter space of rotations and scales. 

5 Since searching the space of rotation and scales in the . 
frequency or space domain is quite complicated (as e.g. 
described in the WO 96/36163), the log-polar map is used 
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where these parameters are Cartesian coordinates and can 
be searched using efficient correlation techniques . 
Resistance to change in aspect ratio: 

Similarly as above, reading the PST template 
in log-log space allows to detect and measure the hori- 
zontal and vertical scaling that was applied to the im- 
age or video frame. This information can then be used to 
compensate the transformation, which the allows the wa- 
termark to be read. 

The use of the log-polar map (LPM) or log-log 
map (LLM) changes depending on whether the watermark was 
inserted block by block of predefined size in the FFT do- 
main or whether the block size depends on the image size. 
In the first case, the LPM or LLM is used to detect scale 
changes' in the image. In the latter case, the maps are 
used to detect the ratio between the FFT size used in em- 
bedding (which is unknown since the original image size 
is unknown in oblivious watermarking) and the FFT size 
used in extraction, which equals the size of the image in 
which we attempt to extract the watermark. This is impor- 
tant in cases where the image size has changed as a re- 
sult of e.g. cropping or rotation since the relative po- 
sitions of the FFT points change. 

Lossy compression : 

The robustness of the watermark to operations 
such as lossy compression is achieved by using a percep- 
tually adaptive spread spectrum communications approach, 
in which a spread spectrum signal is embedded in selected 
components of the magnitude spectrum of the Fourier 
Transform of the image. 

Redundancy: 

The watermark is embedded in blocks of a 
fixed size with exactly the same watermark embedded in 
each block. This means that the watermark can be recov- 
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ered from a single block only. This leads to a redundancy 
that increases the chance of extracting the watermark 
correctly from more than one block. 

5 V . Summary 

The following summarizes some of the proper- 
ties of the preferred embodiments of the invention. 

The use of an asymmetric cryptographic key 
10 pair for the seed generation enables the execution of 

asymmetric key agreement protocols with message recovery 
.or appendix and the protection of the communication be- 
tween the involved parties. Different security services 
for the communication, such as mutual authentication, in- 
15 tegrity, confidentiality and non-repudiation are sup- 
■ - ported by the system with one asymmetric cryptographic 
key pair of the watermark author only for a registration 
or trading process 

The present technique enables a strong bind- 
20 ing relation between the image ID, the image, and the CH 
if the CH registers his copyright at the CCC . If an image 
is watermarked later by an unauthorized person, the time 
stamp in the copyright certificates resolves the copy- 
right ownership. 
25 The CH does not have to reveal his private 

cryptographic key if ownership verification has to be ap- 
plied by a different legal party. 

The present technique supports transferal of 
copyrights. If copyright is transferred to another legal 
30 party, corresponding copyright revocation certificates 
may be generated. 

Digital signatures techniques are applied for 
the security of the communication between different par- 
ties and the authentication data embedded in a private or 
35 public watermark of an image or video. No signature la- 
beling techniques of the complete image or video are ap- 
plied by the system. 
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In addition, originality protection and image 
content protection by ciphering/deciphering in the trans- 
form domain is supported. 

The Fourier Mellin transform is the Fourier 
5 Transform of a log-polar map. It allows to embed a wa- 
termark in a domain that is invariant to rotation, scale 
and translation. However this approach is costly and dif- 
ficult to implement, and therefore it has been enhanced 
by combining with a Fourier Transform based template em- 
10 bedding technique. 

In the present invention, the log-polar map 
of a Fourier transform is used as a means of facilitating 
rotation and scaling invariance. In order to be invariant 
to scaling and change of proportion, the log-log map of 
15 the Fourier transform is also used. 

Circular translation invariants are used as a 
means of constructing digital watermarks that are invari- 
ant to cropping. 

In contrast to some known techniques, the 
20 present system does not require a database of all water- 
marks that were ever embedded in image anywhere . 

Information is embedded and/or retrieved in 
the log-polar or log-log domain of the Fourier transform. 
Frequency components are modulated which are oblivious to 
25 the cover image but which also have the property that 
they form an unambiguous non-repeated pattern in log- 
polar respectively log-log space. They are used for de- 
termining the degree of rotation and scaling respectively 
the change of proportion suffered by a stego-image in the 
30 absence of the cover- image. Coprime frequencies are use- 
ful for generating such a pattern or template. Uniform 
random sampling of log-polar or log-log space is another 
method that can be applied. 

The technique applies a new concept of in- 
35 variants which eliminate the need for explicitly search- 
ing for rotation and/or scaling values. 
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The methods described above can be incorpo- 
rated into an apparatus, such as one or more computers, 
using know programming and hardware techniques. To prove 
the feasibility of the approach, a Java based copyright, 
protection and authentication environment for digital im- 
ages has been implemented. The PKI , the CH, the CCC, and 
the IB application processes all implement a Graphical 
User Interface and a server, supporting both console us- 
ers and other requests through a socket interface. 

While there are shown and described presently 
preferred embodiments of the invention, it is to be dis- 
tinctly understood that the invention is not limited 
thereto but may be otherwise variously embodied and prac- 
ticed within the scope of the following claims. 
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Claims 

1. A method for generating and transmitting a. 
data set between two parties H and B comprising the steps 
5 of 

a) providing a cover data set (CD) corre- 
sponding to the data set to be transmitted, 

b) generating a stego data set (SD) of said 
cover data 'set (CD) by embedding at least one digital wa- 

o termark in said cover data set (CD) , wherein said water- 
mark is encoded using at least one key of an asymmetric 
cryptographic key pair (ps^, vs^) of H, said key pair 
comprising a secret private key (ps H ) and a known public 
key (vspj) derived therefrom, 

5 c) encrypting said stego data set (SD) using 

said key pair (ps^/ vsh) of H, 

d) transmitting said encrypted stego data set 
from said party H to said party B. 

0 2. The method of claim 1, wherein said step 

c) comprises 

generating a mask message (B| | SN) , 
generating a signature (DSSMR G (ps H , B||SN)) 

of said mask message (B| | SN) using said secret private 
5 key (ps H ) , and 

using said signature of said mask message for 

seeding an encryption algorithm for said stego data set 

(SD) . 

0 3 . The method of claim 2 wherein said signa- 

ture (DSSMRq(ps H/ B||SN)) of said mask message (B||SN) 
is transmitted from H to B. 

4 . The method of one of the claims 2 or 3 
5 wherein said encryption algorithm comprises the step of 
calculating the Fourier transform of said stego data set 
(SD), modifying the phase components of the Fourier 
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transform using a pseudo-random pattern seeded by said 
signature (DSSMR G (ps H , B||SN)) of said mask message 
(B||SN) and calculating the inverse Fourier transform for 
generating the encrypted stego data set. 

5 . The method of one of the preceding claims 
wherein said key pair (ps H , vs H ) of H is an elliptic 
curve key pair . 

6. The method of one of the preceding claims 
wherein said step b) further comprises the step of 
generating at least a first watermark, wherein said first 
watermark is encoded using said private key (ps H ) of H. 



7. The method of claim 6 wherein said first 
watermark is encoded using a hash value (crh(ps H )) of 
said private key (ps H ) and can be decoded by using said 
hash value (crh(ps H ) ) . 

8. The method of claim 6 wherein said first 
watermark is encoded using a hash value (crh(OAD CD )) of a 
signature (OAD CD ) generated using said private key (ps H ) . 

25 9. The method of one of the preceding claims 

wherein said step b) further comprises the step of gener- 
ating at least one second watermark, wherein said second 
watermark comprises a pay load <pc H [AM]) derived from the 
Fourier transform of said stego data (SD) . 

30 

10. The method of one of the preceding claims 
wherein said step b) comprises the steps of: 

i) providing a message (s^, S2/ s^) to 

be transmitted in . said at least one watermark, said mes- 
35 sage consisting of a plurality of symbols, 
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ii) providing a pseudo random generator 
seeded with a seed value derived from at least one key of 
said key pair (ps^, vs h) °f H or a h" as h value thereof, 

iii) encoding said message using values from 
5 said pseudo random generator 

iv) using the said encoded message (m) for 
embedding said watermark. 

11. The method of claim 10 wherein said step 
0 iii) comprises: 

for each of said symbols (s^), generating a 
pseudo random sequence of numbers (V2, V2 , ...) by a said 
pseudo random generator, 

using the value of each said symbols (s±) for 
5 selecting a sub-sequence within said pseudo random se- 
quence for forming a symbol vector (r^), and 

adding said symbol vectors (r-jj to generate 
said encoded message (m) . 

12. The method of claim 11 comprising the 
following steps for decoding said message: 

extracting a read-out message (m' ) from said 
watermark, said read-out message being a vector having 
the same length, if erased elements are replaced by zero, 
as said symbol vectors (r-jj, 

generating all possible values of said symbol 
vectors (r-j_) using said pseudo random generator seeded 
with. said seed, and 

calculating the cross-correlation between 
said pseudo random sequences of numbers (v^, V2 , ...) and 
said read-out message (m' ) for retrieving said symbols 
(s ± ) . 

13 . The method of claim 10 wherein said step 
iii) comprises; 

for each bit (bj ) of said symbol sequence 
( s l' s 2' • * * ' s m) ' deriving pseudo random vectors (r-j*) 
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having- elements 1 or -1 from a said pseudo random genera- 
tor, which pseudo random generator preferably generates 
m-sequences or Gold codes, and 

depending on the value of said bit (bj), mul- 
tiplying said pseudo random vector (rj*) with +1 or -1 to 
generate a modified pseudo random vector, and adding said 
modified pseudo random vectors to generate an encoded 
message (m) . 

14. The method of claim 13 comprising the 
following steps for decoding said message: 

extracting a read-out message (m' ) from said 

watermark, 

deriving said pseudo random vectors (rj ) 
15 from said pseudo random generator seeded with a said 
seed, and 

calculating the cross correlation between 
each of said pseudo random vectors (rj*) and said read-, 
out message <m' ) for retrieving the corresponding bit 
20 (bj) of the said symbol sequence (si, S2 / . s M ) . 

15. The method of one of the claims 10 - 14 
wherein the position of components to be modulated by 
each value of the encoded message (m) is given by a 

25 pseudo random generator seeded by a key known by both H 
and B . 

16. The method of one of the preceding claims 
comprising the step of encoding a message for being em- 

30 bedded in said watermark by using symbol based Reed Solo- 
mon codes as error control codes. 

17. The method of one of the preceding claims 
wherein said step b) further comprises the step of calcu- 

35 lating a logarithm of said cover data set (CD) before em- 
bedding said watermark for embedding said watermark in a 
perceptually flat domain. 
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18. A method for generating a stego data set 
(SD) from a cover data set (CD) especially for step b) of 
one of the preceding claims, comprising the steps of: 

generating at least one message (ID CD ) , 

digitally signing said message (IDq D ) using 
an asymmetric cryptographic key pair (ph* v h> anc ^ a sig- 
nature generating algorithm (DSSMR) with message recovery 
for generating a digital signature (OADqq) , and 

generating said stego data set (SD) of said 
cover data set (CD) by generating at least one digital 
watermark, wherein said digital signature (OADq^) is used 
for deriving a seed for generating said watermark. 

19 . Method for generating and verifying a wa- 
termark in a cover data set (CD) representing a two- 
dimensional cover image, especially for step b) of one of 
the preceding claims, comprising the following steps for 
generating said watermark 

A) calculating the Fourier transform of at 
least part of cover data set (CD) for generating Fourier 
components of said cover image, and 

B) modulating at least part of said Fourier 
components using a template modulation pattern (T' ) , 

C) using the inverse Fourier transform for 
generating a stego data set (SD) , 

said method further comprising the following steps for 
verifying said watermark in a possibly scaled and/or ro- 
tated version of said stego data set (SD) , 

D) calculating the Fourier transform of the 
possibly scaled and/or rotated version of said stego data 
set (SD) for generating Fourier components of said stego 
data set, 

E) calculating a log-polar or log-log trans- 
form of said Fourier components of said stego data set 
(SD) , and 
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F) calculating the cross correlation between 
a log-polar or log-log transform (T) of said modulation 
pattern (T' ) and said log-polar or log-log transform of 
said Fourier components of said stego data set for evalu- 
5 ating a scaling and/or rotation factor. 

20. The method of claim 19 wherein said step 
B) further comprises the steps of 

calculating a log-polar or log-log transform 
of said components of said cover data set for generating 
log-polar components , 

modulating said log polar components using a 
log-polar or log-log transform (T) of said modulation 
pattern (T' ) . 

21. A method for verifying a watermark in a 
possibly rotated and/or scaled version of a two or three 
dimensional stego data set (SD), comprising the steps of: 

A) calculating a Fourier transform of said 
stego data set (SD), 

B) calculating a log-polar or a log-log 
transform of said Fourier transform of said stego data 
set , 

C) calculating the correlation between said 
log-polar or log-log transform and a template (T) , which 
template is the log-polar or log-log transformation of 
said watermark. 

22. The method of claim 21, wherein said step 
30 B) comprises the step of calculating the log-polar trans- 
form of said Fourier transform of said stego data set and 
said step C) comprises a step of detecting a rotation and 
either a uniform scaling suffered by said stego data set 
or a ratio between block size used in embedding and ex- 

35 traction of said watermark. 
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23. The method of claim 21, wherein said step 
B) comprises the step of calculating the log-log trans- 
form of said Fourier transform of said stego data set and 
said step C) comprises a step of detecting either a 

5 change in aspect ratio suffered by said stego data set or 
a change of aspect ratio between block sizes used in em- 
bedding and extraction of said watermark. 

24. The method of claim 21, wherein the pres- 
et ence of said watermark is verified by means of a Bayesian 

approach to detect the presence of said watermark given a 
said key without decoding said watermark. 

25. The method of one of the claims 21-24, 
5 further comprising at least one of the following steps: 

i) pre- filtering said cover data by applying 
a windowing algorithm thereto, preferably Blackman, Han- 
ning or Welch windowing, and/or 

ii) calculating the variance or distribution 
of the Fourier transform locally for filtering outliers 
and noise, and/or 

iii) locating local peaks in said Fourier 
transform and carrying out said step B) for these local 
peaks only, preferably transforming only the coordinates 
of these local peaks, and preferably using the log- log or 
log-polar transform of said coordinates for calculating 
said correlation, 

iv) excluding low frequency data from said 
Fourier transform before carrying out said step B) , 
and/ or, 

v) detecting a scaling and/or rotation in 
said step C) , using said scaling and/or rotation for ei- 
ther a) scaling and/or or rotating said Fourier trans- 
form, calculating a scaled and/or rotated log-log or log- 
polar transform therefrom and correlating said rotated 
log-log or log-polar transform with said template, or b) 
calculating a second template by scaling and/or rotating 
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an original Fourier-space template and calculating a log- 
log and or log-polar transform therefrom and using said 
second template for calculation a second correlation with 
said log-log or log-polar transform of said stego data, 
5 and/ or 

vi) weighing low frequency components of said 
log-log or log-polar transform stronger that high fre- 
quency components while carrying out said correlation. 

10 ' 26. A method for generating a stego data set 

(SD) from a cover data set (CD) especially for step b) of 
one of the claims 1-18, comprising the step of modulat- 
ing said cover data set (CD) using a given pattern, which 
pattern is calculated from a watermark using the follow- 

15 ing steps : 

A) providing said watermark, 

B) calculating a first inverse Fourier trans- 
form of said watermark, 

C) calculating an inverse log-log or log- 
20 polar, transform of said watermark, and 

D) calculating said pattern from said inverse 
log-log or log-polar transform. 

27. The method of claim 2 6 further comprising 
the step of combining the magnitude components of said 
first inverse Fourier transform with the phases of a Fou- 
rier transform of said stego data (SD) to generate a fre- 
quency space pattern and, preferably, calculating a sec- 
ond inverse Fourier transform of said frequency space 
pattern. 

28. A method for verifying a watermark in a 
possibly rotated and/or. scaled version of a two or three 
dimensional stego data set (SD), preferably as generated 

35 in one of the claims 25 or 26, comprising the steps of: 

A) calculating a first Fourier transform of 
said stego data set (SD) , 
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B) calculating a log-polar or a log-log 
transform of said Fourier transform of said stego data 
set , 

C) calculating a second Fourier. transform of 
5 said log-polar or log-log transform and searching said 

watermark in said second. Fourier transform. 

29. A method for generating a watermark in a 
cover data set (CD) representing a two or three dimen- 
sional data set, especially for step b) of one of the 
preceding claims, comprising the following steps: 

A) generating a template modulation pattern 
(TV) using a random number generator seeded by a key (K) , 

B) calculating the Fourier transform of at 
least part of said cover data set (CD) for generating 
Fourier components of said cover data set, 

C) modulating at least part of said Fourier 
components using said template modulation pattern ( T ' ) , 

D) using the inverse Fourier transform for 
generating a stego-image. 

30. Method for generating a watermark in a 
cover data set (CD) representing a cover image especially 
for one of the preceding claims, characterized by the 

25 step of dividing said image into a plurality of blocks 
and by the following steps carried out for each block: 

i) calculating the Fourier transform of the 

block, 

ii) modulating at least part of the magnitude 
30 components of the Fourier transform of the block using a 

modulation pattern, which modulation pattern defines val- 
ues to be added/ subtracted to/ from said magnitude compo- 
nents, wherein for each magnitude component its corre- 
sponding phase component determines if said value is to 
35 be added or subtracted, and wherein the same modulation 
pattern is used for all blocks . 
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31. The method of claim 30 wherein said 
blocks are adjacent. 

32. The method of claim 30 wherein the said 
5 image is divided into a plurality of overlapping blocks 

and wherein the step i) comprises calculating the Lapped 
Orthogonal transform of each block to embed a Lapped Or- 
thogonal transform based watermark. 

10 33. The method of claim 30 wherein the said 

image is divided into a plurality of non-square blocks 
and wherein the step i) consists in padding each block 
with appropriate values (constant or symmetric extension) 
in order to obtain square blocks, calculating the Fou- 

15 rier transform of each obtained square block to embed 
Fourier transform based watermark. 

34. The method of claim 30 wherein the said 
image is divided into a plurality of non-square blocks 

20 and wherein said step i) comprises computing the arbi- 
trary length wavelet transform of each block to embed a 
wavelet transform based watermark. 

35, The method of one of the claims 30 - 34 
25 wherein the watermark is applied to all or some of the 

frames of a video. 



36. A method for generating a stego data set 

<SD) from a cover data set (CD) especially for step b) of 
30 one of the claims 1-18, by adding a watermark to said 

cover data set, wherein said cover data set comprises 

video data, comprising the steps of 

generating three dimensional spatio-temporal 

blocks of said video data and 
3 5 applying said watermark to each of said 

blocks, preferably by calculating a Fourier transform of 

each of said blocks . 



WO 99/17536 



PCT/IB98/01500 



62 

37. A method for generating a stego data set 
(SD) from a cover data set (CD) especially according to 
one of the preceding claims, by adding a watermark to 
said cover data set comprising the steps of 

dividing said stego data sets into blocks, 
calculating a lapped orthogonal transform 
(LOT) of each of said blocks, and 

applying said watermark to said lapped or- 
thogonal transforms . 

38. The method of claim 37 further comprising 
the step of modulating selected components of said lapped 
orthogonal transform (LOT) as a function of a local image 
characteristics, such as the local image variance. 

39. Method for generating and transmitting a 
data set between two parties H and B, especially of one 
of the preceding claims, comprising the steps of 

providing a cover data set (CD) corresponding 
to the data set to be transmitted, 

generating a stego data set (SD) of said 
cover data set (CI) at a party H by generating at least 
one digital watermark in said cover data set (CD) , 

transmitting a hash value of said stego data 
set (3D) to a registration party (O) , and 

permanently storing certification data (CCD) 
at said registration party (O) , said certification data 
comprising said hash value of said stego data set (SI) , a 
digital time stamp (TVP) and information designating said 
party H. 

40, The method of claim 39 further comprising 
the steps of generating a digital signature of said cer- 
tification data (CCD) using an asymmetric cryptographic 
key pair (psg, vsq) of said registration party (0) , 
transmitting said certification data (CCD) and said digi- 
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tal signature to said party H, and verifying said digital 
signature at said party H by using a public key (vs G ) of 
said key pair of said registration party. 

5 41. A method for embedding a watermark in a 

cover data set for generating a stego data set, espe- 
cially of one of the preceding claims, comprising the. 
steps of 

calculating at least some magnitude Fourier 
o components (MC) of said cover data set (CD) , 

applying an authentication function (AF) for 
-generating a value (AM) derived from said Fourier compo- 
nents (MC) , 

ciphering said value (AM) using a secret key 
5 (pc„) of an asymmetric key pair (pc H , vc„) for generating 
a ciphered message, and 

embedding said ciphered message as a payload 

in a public watermark. 

0 42. A method for verifying the originality of 

a possibly modified stego data set generated with the 
method of claim 41 comprising the step of reading said 
value (AM) by decoding said ciphered message using the 
public key of said key par and comparing said magnitude 

:5 Fourier components to said stego data set. 
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